Generative Probabilistic Models for Retrieval of Documents with Structure and Annotations

Introduction Structure in documents has been a part of Information Retrieval as long as the field has existed. There have always been documents with metadata information such as authors and creation dates. Researchers have always recognized that document structure is important to effective retrieval. The earliest systems supporting structure focused on providing support for querying of the fielded information. This has strong connections to research in library sciences, where fielded search of citations is important. A common approach to handling fielded queries was to treat constraints literally, perhaps providing a ranking of the query corresponding to how well the multiple constraints are met. A typical example of an early citation search system was the Norton Cotton Cancer Center (NCCC) On-Line Personal Bibliographic Retrieval System [10]. The bibliographic system stored citations of documents (articles, books, journals, etc.) in a fielded database. Citations were indexed with keywords in a controlled vocabulary or general language. Titles, authors, publication information, and call numbers were also indexed in fields. Searches could query the entire citation or over the controlled vocabulary. The searches were either conjunctions or disjunctions of the clauses; more sophisticated search capabilities were not deemed necessary. Results were ordered by the author field. A more sophisticated example of citation search is the SCAT-IR system [37]. SCAT-IR indexed similar fields as the NCCC bibliographic system, but allowed additional query structures. Each field could be queried in a clause, and query clauses could be combined with Boolean ANDs and ORs. Yet result sets were not ordered by any estimate of relevance. Fox summarized early work with structured documents in [11]. Much work to that date was empirical, with no analysis of the conditions in which structure is informative. Most tasks limited the retrieval unit to documents, although some early work with passage retrieval had been performed. Fox described some experiments that found a vector space system performed better on sections than on passages, but performed no analysis as to why. Fox further described how soft Boolean matching functions such as the P-Norm formalism can be extended to matching in complex documents. Up to this point there was little consideration of what the best unit of retrieval is and almost no consideration of relationship between fields of information in a document. The introduction of Inference Networks by Turtle and Croft [50] provided one of the first retrieval models explicitly designed to handle multiple representations of information …